[Inference API] Add Tencent Cloud AI Service to Inference API#152829
Open
boicehuang wants to merge 36 commits into
Open
[Inference API] Add Tencent Cloud AI Service to Inference API#152829boicehuang wants to merge 36 commits into
boicehuang wants to merge 36 commits into
Conversation
elasticsearchmachine
added
needs:triage
dimitris-athanasiou
added
:SearchOrg/Inference
Collaborator
|
Pinging @elastic/search-inference-team (Team:Search - Inference) |
# Conflicts: # x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/InferenceNamedWriteablesProvider.java
Contributor
|
@elasticmachine test this please |
Contributor
|
@boicehuang looks like compiling is failing with this: If you could take a look that'd be great. |
Contributor
|
Hey @boicehuang looks like there are some failing tests: Here's an example of how to run one of them: If you could take a look at fixing them that'd be great. |
…centcloud_inference
Contributor
Author
|
@jonathan-buttner all tests are passed |
Contributor
|
@boicehuang could you take a look at this failure: |
Contributor
Author
|
@jonathan-buttner i have fix it. please retry the cli test. |
jonathan-buttner
requested changes
Jul 17, 2026
jonathan-buttner
left a comment
jonathan-buttner
left a comment
Contributor
There was a problem hiding this comment.
Thanks for addressing the tests, I left a few suggestions.
| @@ -0,0 +1,5 @@ | |||
| pr: 152829 | |||
| summary: Add Tencent Cloud inference service | |||
| area: Machine Learning | |||
Contributor
There was a problem hiding this comment.
Suggested change
| area: Machine Learning | |
| area: Inference |
| */ | ||
| @Override | ||
| public ExecutableAction accept(TencentCloudActionVisitor visitor, Map<String, Object> taskSettings) { | ||
| throw new UnsupportedOperationException( |
Contributor
There was a problem hiding this comment.
ESQL leverages the completion task type which typically uses the class like this one via the visitor pattern. We should implement the completion functionality so ESQL can leverage this integration.
Another option instead of the visitor pattern is to use a strategy like ElasticInferenceService:
https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/elastic/action/ElasticInferenceServiceActionCreator.java#L43
| * Dispatches TencentCloud AI Gateway chat completion requests. TencentCloud is OpenAI-compatible so the OpenAI | ||
| * response handlers are reused. | ||
| */ | ||
| public class TencentCloudChatCompletionRequestManager extends BaseRequestManager { |
Contributor
There was a problem hiding this comment.
We're trying to move away from additional *RequestManager classes. Here's how OpenAI does it: https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/openai/OpenAiService.java#L245-L251
Can we follow that pattern for doUnifiedCompletionInfer or follow what the ElasticInferenceService does and then remove this class?
| import java.util.Map; | ||
| import java.util.Objects; | ||
|
|
||
| public class TencentCloudChatCompletionServiceSettings extends FilteredXContentObject |
Contributor
There was a problem hiding this comment.
We're refactoring how we parse settings. Let's move to the new way which is to use an ObjectParser. Take a look at https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/llama/embeddings/LlamaEmbeddingsServiceSettings.java for an example.
| int initialValidationErrorCount = validationException.validationErrors().size(); | ||
|
|
||
| var modelId = extractRequiredString(map, ServiceFields.MODEL_ID, ModelConfigurations.SERVICE_SETTINGS, validationException); | ||
| var uri = extractOptionalUri(map, ServiceFields.URL, validationException); |
Contributor
There was a problem hiding this comment.
Can users change the URL (like can they spin up deployments)? Or will this always point to the same place? If it's the same place then let's hard code it instead like JinaAI: https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/jinaai/rerank/JinaAIRerankModel.java#L31-L33
|
|
||
| @Override | ||
| public TransportVersion getMinimalSupportedVersion() { | ||
| return TransportVersion.minimumCompatible(); |
Contributor
There was a problem hiding this comment.
We'll need to create a transport version and reference in places like this: https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/elastic/completion/ElasticInferenceServiceChatCompletionTaskSettings.java#L177
| // Batch limit for embedding chunking. TencentCloud AI Gateway does not document a hard cap; use a conservative value. | ||
| private static final int EMBEDDING_MAX_BATCH_SIZE = 32; | ||
|
|
||
| private static final EnumSet<TaskType> SUPPORTED_TASK_TYPES = EnumSet.of( |
Contributor
There was a problem hiding this comment.
Let's add TaskType.COMPLETION
|
|
||
| @Override | ||
| public int rerankerWindowSize(String modelId) { | ||
| // BGE reranker models (bge-reranker-large, bge-reranker-v2-m3) support up to 512-token inputs. |
Contributor
There was a problem hiding this comment.
Are these the only rerank models that are supported?
|
|
||
| // Empty default task settings | ||
| namedWriteables.add(new NamedWriteableRegistry.Entry(TaskSettings.class, EmptyTaskSettings.NAME, EmptyTaskSettings::new)); | ||
| namedWriteables.add( |
Contributor
There was a problem hiding this comment.
This is probably a merge conflict resolution issue, let's add these changes back and we need to follow a similar pattern as the other services in this file to add the named writeables that are introduced in this PR.
| import static org.mockito.Mockito.mock; | ||
| import static org.mockito.Mockito.when; | ||
|
|
||
| public class TencentCloudServiceTests extends InferenceServiceTestCase { |
Contributor
There was a problem hiding this comment.
Let's follow the same pattern for OpenAi here. Take a look at: https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/OpenAiServiceSettingsTests.java
Also take a look at the other tests in that directory: https://github.com/elastic/elasticsearch/tree/main/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai we'll want to add similar ones
…on, COMPLETION support, visitor pattern - Register all TencentCloud ServiceSettings/TaskSettings in InferenceNamedWriteablesProvider - Add dedicated TransportVersion (ml_inference_tencentcloud_added, ID 9479000) - Add TaskType.COMPLETION support for ESQL - Remove TencentCloudChatCompletionRequestManager, route completion via visitor pattern - Fix changelog area: Machine Learning -> Inference - Update tests for streaming tasks and COMPLETION
Replace manual map field extraction with declarative ObjectParser parsing in all TencentCloud service settings classes, following the pattern set by LlamaEmbeddingsServiceSettings. Common fields (model_id, url, rate_limit, api_key) are declared once via TencentCloudCommonServiceSettings.declareCommonFields() and reused by each task-specific settings parser. - TencentCloudCommonServiceSettings: add declareCommonFields(), CommonSettingsBuilder interface, ObjectParser-based fromMap() - TencentCloudEmbeddingsServiceSettings: use ObjectParser for common + embeddings-specific fields (similarity, dimensions, max_input_tokens) - TencentCloudChatCompletionServiceSettings: use ObjectParser with chat-completion default rate limit - TencentCloudRerankServiceSettings: use ObjectParser for common fields
…loud service
Users cannot bring their own TencentCloud deployment, but they may need to
target different regional endpoints. Replace the free-form 'url' setting with
a 'region' setting (default 'bj'). The endpoint URL is always constructed as
https://{region}.aisearch.tencentelasticsearch.com/v1/<task-path>.
- TencentCloudUtils: add buildHost(region), buildUri(region, path...) helpers
- TencentCloudCommonServiceSettings: replace @nullable URI uri with String region
- All model classes: resolveUri() now builds URI from region, no url override
- TencentCloudService: add usesParserForServiceSettings() override, replace
URL config with region config
- Remove URL from declareCommonFields (consumed silently for backward compat)
- Tests: rewrite URL-related tests to use region; remove WebServer-dependent
integration tests (endpoint is now hardcoded to *.tencentelasticsearch.com)
…oud service - Update rerankerWindowSize() comment: rerank supports multiple models (not a fixed list), model IDs are passed through to the gateway - Update MODEL_ID config description to mention the full list is in gateway documentation
- Delete TencentCloudEndpointUtils.java (unused after URL→region migration) - Make COMPLETION_ERROR_PREFIX and USER_ROLE private in ActionCreator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Tencent Cloud provides an OpenAI-compatible AI Gateway that unifies access to a
family of Chinese and multilingual models used with Tencent Cloud
Elasticsearch. It exposes three endpoints, all authenticated via
Authorization: Bearer sk-<api-key>:POST /v1/embeddingstext_embeddingbge-base-zh-v1.5,bge-large-zh-v1.5,bge-m3,Conan-embedding-v1,Qwen3-Embedding-0.6B, …POST /v1/chat/completionschat_completion/completiondeepseek-v3,deepseek-r1POST /v1/rerankrerankbge-reranker-large,bge-reranker-v2-m3